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(57) ABSTRACT 

An optical mouse images as an array of pixels the spatial 
features of generally any micro textured or micro detailed 
work surface below the mouse. The photo detector responses 
are digitized and stored as a frame into memory. Motion 
produces successive frames of translated patterns of pixel 
information, which are compared by autocorrelation to 
ascertain the direction and amount of movement. A hold 
feature suspends the production of movement signals to the 
computer, allowing the mouse to be physically relocated on 
the work surface without disturbing the position on the 
screen of the pointer. This may be needed if the operator runs 
out of room to physically move the mouse further, but the 
screen pointer still needs to go further. The hold feature may 
be implemented with an actual button, a separate proximity 
detector or by detecting the presence of a characteristic 
condition in the digitized data, such as loss of correlation or 
velocity in excess of a selected limit. A convenient place for 
an actual hold button is along the sides of the mouse near the 
bottom, where the thumb and the opposing ring finger grip 
the mouse. The gripping force used to lift the mouse engages 
the hold function. Hold may incorporate a brief delay upon 
either the release of the hold button, detection of proper 
proximity or the return of reasonable digitized values. Dur- 
ing that delay any illumination control or AGC servo loops 
stabilize. A new reference frame is taken prior to the 
resumption of motion detection. 

1 Claim, 7 Drawing Sheets 
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PROXIMITY DETECTOR FOR A SEEING 
EYE MOUSE 

REFERENCE TO RELATED PATENTS 

This Application is related to the subject matter described 5 
in the following two U.S. Pat. No. 5,578,813 filed Mar. 2, 
1995, issued Nov. 26, 1996 and entitled FREEHAND 
IMAGE SCANNING DEVICE WHICH COMPENSATES 
FOR NON-LINEAR MOVEMENT; and U.S. Pat. No. 
5,644,139, filed Aug. 14, 1996, issued Jul 1, 1997 and 
entitled NAVIGATION FOR DETECTING MOVEMENT 
OF NAVIGATION SENSORS RELATIVE TO AN 
OBJECT. Both of these Patents have the same inventors: 
Ross R. Allen, David Beard, Mark T. Smith and Barclay J. 
Tullis, and both Patents are assigned to Hewlett-Packard Co. 
This application is also related to the subject matter 
described in U.S. Pat. No. 5,786,804 filed Oct. 6, 1995, 
entitled METHOD AND SYSTEM FOR TRACKING 
ATTITUDE, issued Jul. 28, 1998, and also assigned to 2Q 
Hewlett-Packard Co. These three Patents describe tech- 
niques of tracking position movement. Those techniques are 
a component in the preferred embodiment described below. 
Accordingly, U.S. Pat. Nos. 5,578,813, 5,644,139 and 5,786, 
804 are hereby incorporated herein by reference. 25 

BACKGROUND OF THE INVENTION 

The use of a hand operated pointing device for use with 
a computer and its display has become almost universal. By 
far the most popular of the various devices is the conven- 30 
tional (mechanical) mouse. A conventional mouse typically 
has a bottom surface carrying three or more downward 
projecting pads of a low friction material that raise the 
bottom surface a short distance above the work surface of a 
cooperating mouse pad. Centrally located within the bottom 35 
surface of the mouse is a hole through which a portion of the 
underside of a rubber-surfaced steel ball (hereinafter called 
simply a rubber ball) extends; in operation gravity pulls the 
ball downward and against the top surface of the mouse pad. 
The mouse pad is typically a closed cell foam rubber pad 40 
covered with a suitable fabric. The low friction pads slide 
easily over the fabric, but the rubber ball does not skid, but 
instead rolls as the mouse is moved. Interior to the mouse are 
rollers, or wheel, that contact the ball at its equator (the great 
circle parallel to the bottom surface of the mouse) and 45 
convert its rotation into electrical signals. The external 
housing of the mouse is shaped such that when it is covered 
by the user's hand it appears to have a "front-to-back" axis 
(along the user's forearm) and an orthogonal "left-to-right" 
axis. 'The interior wheels that contact the ball's equator are 50 
arranged so that one wheel responds only to rolling of the 
ball that results from a motion component of the mouse that 
is along the front-to-back axis, and also so that the other 
wheel responds only to rolling produced by a motion com- 
ponent along the left-to-right axis. The resulting rotations of 55 
the wheels or contact rollers produce electrical signals 
representing these motion components. (Say, F/B represent- 
ing Forward and Backward, and L/R representing Left or 
Right.) These electrical signals F/B and L/R are coupled to 
the computer, where software responds to the signals to 60 
change by a Ax and a Ay the displayed position of a pointer 
(cursor) in accordance with movement of the mouse. The 
user moves the mouse as necessary to get the displayed 
pointer into a desired location or position. Once the pointer 
on the screen points at an object or location of interest, one 65 
of one or more buttons on the mouse is activated with the 
fingers of the hand holding the mouse. The activation serves 
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as an instruction to take some action, the nature of which is 
defined by the software in the computer. 

Unfortunately, the usual sort of mouse described above is 
subject to a number of shortcomings. Among these are 
deterioration of the mouse ball or damage to its surface, 
deterioration or damage to the surface of the mouse pad, and 
degradation of the ease of rotation for the contact rollers 
(say, (a) owing to the accumulation of dirt or of lint, or (b) 
because of wear, or (c) both (a) and (b)). All of these things 
can contribute to erratic or total failure of the mouse to 
perform as needed. These episodes can be rather frustrating 
for the user, whose complaint might be that while the cursor 
on the screen moves in all other directions, he can't get the 
cursor to, say, move downwards. Accordingly, industry has 
responded by making the mouse ball removable for easy 
replacement and for the cleaning of the recessed region into 
which it fits. Enhanced mouse ball hygiene was also a prime 
motivation in the introduction of mouse pads. Nevertheless, 
some users become extremely disgusted with their particular 
mouse of the moment when these remedies appear to be of 
no avail. Mouse and mouse pad replacement is a lively 
business. 

The underlying reason for all this trouble is that the 
conventional mouse is largely mechanical in its construction 
and operation, and relies to a significant degree on a fairly 
delicate compromise about how mechanical forces are 
developed and transferred. 

There have been several earlier attempts to use optical 
methods as replacements for mechanical ones. These have 
included the use of photo detectors to respond to mouse 
motion over specially marked mouse pads, and to respond to 
the motion of a specially striped mouse ball. U.S. Pat. No. 
4,799,055 describes an optical mouse that does not require 
any specially pre-marked surface. (Its disclosed two 
orthogonal one pixel wide linear arrays of photo sensors in 
the X and Y directions and its state-machine motion detec- 
tion mechanism make it a distant early cousin to the tech- 
nique of the incorporated Patents, although it is our view that 
the shifted and correlated array [pixel pattern within an area] 
technique of the incorporated Patents is considerably more 
sophisticated and robust.) To date, and despite decades of 
user frustration with the mechanical mouse, none of these 
earner optical techniques has been widely accepted as a 
satisfactory replacement for the conventional mechanical 
mouse. Thus, it would be desirable if there were a non- 
mechanical mouse that is viable from a manufacturing 
perspective, relatively inexpensive, reliable, and that 
appears to the user as essentially the operational equivalent 
of the conventional mouse. This need could be met by a new 
type of optical mouse has a familiar "feel" and is free of 
unexpected behaviors. It would be even better if the opera- 
tion of this new optical mouse did not rely upon cooperation 
with a mouse pad, whether special or otherwise, but was 
instead able to navigate upon almost any arbitrary surface. 

SUMMARY OF THE INVENTION 

A solution to the problem of replacing a conventional 
mechanical mouse with an optical counterpart is to optically 
detect motion by directly imaging as an array of pixels the 
various particular spatial features of a work surface below 
the mouse, much as human vision is believed to do. In 
general, this work surface may be almost any flat surface; in 
particular, the work surface need not be a mouse pad, special 
or otherwise. To this end the work surface below the imaging 
mechanism is illuminated from the side, say, with an infrared 
(IR) light emitting diode (LED). A surprisingly wide variety 
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of surfaces create a rich collection of highlights and shadows down), but at some point the attendant complexity ruins the 

when illuminated with a suitable angle of incidence. That advantage, and it is preferable to simply have a sufficiently 

angle is generally low, say, on the order of five to twenty high frame rate with small trial shifts. The trial shift with the 

degrees, and we shall term it a "grazing" angle of incidence. least difference (greatest correlation) can be taken as an 

Paper, wood, formica and painted surfaces all work well; 5 indication of the motion between the two frames. That is, it 

about the only surface that does not work is smooth glass provides a raw F/B and L/R. The raw movement information 

(unless it is covered with fingerprints!).The reason these may be scaled and or accumulated to provide display pointer 

surfaces work is that they possess a micro texture, which in movement information (Ax and Ay) of a convenient granu- 

some cases may not be perceived by the unaided human larity and at a suitable rate of information exchange, 

senses. io The actual algorithms described in the incorporated Pat- 

IR light reflected from the micro textured surface is ents (and used by the seeing eye mouse) are refined and 

focused onto a suitable array (say, 16x16 or 24x24) of photo sophisticated versions of those described above. For 

detectors. The LED may be continuously on with either a example, let us say that the photo detectors were a 16x16 

steady or variable amount of illumination servoed to maxi- array. We could say that we initially take a reference frame 

mize some aspect of performance (e.g., the dynamic range 15 by storing the digitized values of the photo detector outputs 

of the photo detectors in conjunction with the albedo of the as they appear at some time to. At some later time X x we take 

work surface). Alternatively, a charge accumulation mecha- a sample frame and store another set of digitized values. We 

nism coupled to the photo detectors may be "shuttered" (by wish to correlate a new collection of nine comparison frames 

current shunting switches) and the LED pulsed on and off to (thought to be, null, one over, one over and one up, etc.) 

control the exposure by servoing the average amount of 20 against a version of the reference frame representing "where 

light. Turning the LED off also saves power; an important we were last time". The comparison frames are temporarily 

consideration in battery operated environments. The shifted versions of the sample frame; note that when shifted 

responses of the individual photo detectors are digitized to a comparison frame will no longer overlap the reference 

a suitable resolution (say, six or eight bits) and stored as a frame exactly. One edge, or two adjacent edges will be 

frame into corresponding locations within an array of 25 unmatched, as it were. Pixel locations along the unmatched 

memory. Having thus given our mouse an "eye" , we are edges will not contribute to the corresponding correlation 

going to further equip it to "see" movement by performing (i.e., for that particular shift), but all the others will. And 

comparisons with successive frames. those others are a substantial number of pixels, which gives 

Preferably, the size of the image projected onto the photo rise to a very good signal to noise ratio. For "nearest 

detectors is a slight magnification of the original features 30 neighbor" operation (i.e., limited to null, one over, one 

being imaged, say, by two to four times. However, if the up/down, and the combinations thereof) the correlation 

photo detectors are small enough it may be possible and produces nine "correlation values", which may be derived 

desirable to dispense with magnification. The size of the from a summing of squared differences for all pixel locations 

photo detectors and their spacing is such that there is much having spatial correspondence (i.e., a pixel location in one 

more likely to be one or several adjacent photo detectors per 35 frame that is indeed paired with a pixel location in the other 

image feature, rather than the other way around. Thus, the frame— unmatched edges won't have such pairing), 

pixel size represented by the individual photo detectors A brief note is perhaps in order about how the shifting is 

corresponds to a spatial region on the work surface of a size done and the correlation values obtained. The shifting is 

that is generally smaller than the size of a typical spatial accomplished by addressing offsets to memories that can 

feature on that work surface, which might be a strand of fiber 40 output an entire row or column of an array at one time, 

in a cloth covering a mouse pad, a fiber in a piece of paper Dedicated arithmetic circuitry is connected to the memory 

or cardboard, a microscopic variation in a painted surface, or array that contains the reference frame being shifted and to 

an element of an embossed micro texture on a plastic the memory array that contains the sample frame. The 

laminate. The overall size of the array of photo detectors is formulation of the correlation value for a particular trial shift 

preferably large enough to receive the images of several 45 (member of the nearest or near neighbor collection) is 

features. In this way, images of such spatial features produce accomplished very quickly. The best mechanical analogy is 

translated patterns of pixel information as the mouse moves. to imagine a transparent (reference) film of clear and dark 

The number of photo detectors in the array and the frame patterns arranged as if it were a checker board, except that 

rate at which their contents are digitized and captured the arrangement is perhaps random. Now imagine that a 

cooperate to influence how fast the seeing-eye mouse can be 50 second (sample) film having the same general pattern is 

moved over the work surface and still be tracked. Tracking overlaid upon the first, except that it is the negative image 

is accomplished by comparing a newly captured sample (dark and clear are interchanged). Now the pair is aligned 

frame with a previously captured reference frame to ascer- and held up to the light. As the reference film is moved 

tain the direction and amount of movement. One way that relative to the sample film the amount of light admitted 

may be done is to shift the entire content of one of the frames 55 through the combination will vary according to the degree 

by a distance of one pixel (corresponds to a photo detector), that the images coincide. The positioning that admits the 

successively in each of the eight directions allowed by a one least light is the best correlation. If the negative image 

pixel oflset trial shift (one over, one over and one down, one pattern of the reference film is a square or two displaced 

down, one up, one up and one over, one over in the other from the image of the sample film, the positioning admits the 

direction, etc.). That adds up to eight trials, but we mustn't 60 least light will be one that matches that displacement. We 

forget that there might not have been any motion, so a ninth take note of which displacement admits the least light; for 

trial "null shift" is also required. After each trial shift those the seeing eye mouse we notice the positioning with the best 

portions of the frames that overlap each other are subtracted correlation and say that the mouse moved that much. That, 

on a pixel by pixel basis, and the resulting differences are in effect, is what happens within an integrated circuit (IC) 

(preferably squared and then) summed to form a measure of 65 having photo detectors, memory and arithmetic circuits 

similarity (correlation) within that region of overlap. Larger arranged to implement the image correlation and tracking 

trial shifts are possible, of course (e.g., two over and one technique we are describing. 
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It would be desirable if a given reference frame could be 
re-used with successive sample frames. At the same time, 
each new collection of nine (or twenty-five) correlation 
values (for collections at t,-, t 1+1 , etc.) that originates from a 
new image at the photo detectors (a next sample frame) 5 
should contain a satisfactory correlation. For a hand held 
mouse, several successive collections of comparison frames 
can usually be obtained from the (16x16) reference frame 
taken at Iq. What allows this to be done is maintaining 
direction and displacement data for the most recent motion 10 
(which is equivalent to knowing velocity and time interval 
since the previous measurement). 

This allows "prediction" of how to (permanently!) shift 
the collection of pixels in the reference frame so that for the 
next sample frame a "nearest neighbor** can be expected to 15 
correlate. This shifting to accommodate prediction throws 
away, or removes, some of the reference frame, reducing the 
size of the reference frame and degrading the statistical 
quality of the correlations. When an edge of the shifted and 
reduced reference frame begins to approach the center of 20 
what was the original reference frame it is time to take a new 
reference frame. This manner of operation is termed "pre- 
diction" and could also be used with comparison frames that 
are 5x5 and an extended "near neighbor*' (null, two over/one 
up, one over/two up, one over/one up, two over, one over, . 25 
. . ) algorithm. The benefits of prediction are a speeding up 
of the tracking process by streamlining internal correlation 
procedure (avoiding the comparison of two arbitrarily 
related 16x16 arrays of data) and a reduction of the per- 
centage of time devoted to acquiring reference frames. 30 

In addition to the usual buttons that a mouse generally 
has, our seeing eye mouse may have another button that 
suspends the production of movement signals to the 
computer, allowing the mouse to be physically relocated on 
the work surface without disturbing the position on the 
screen of the pointer. This may be needed if the operator runs 
out of room to physically move the mouse further, but the 
screen pointer still needs to go further. This may happen, say, 
in a UNIX system employing a display system known as 
"Single Logical Screen" (SLS) where perhaps as many as 
four monitors are arranged to each display some subportion 
of the overall "screen**. If these monitors were arranged as 
one high by four across, then the left to right distance needed 
for a single corresponding maximal mouse movement would 
be much wider than usually allowed for. The usual maneuver 
executed by the operator for, say, an extended rightward 
excursion, is to simply pick the mouse up at the right side of 
the work surface (a mouse pad, or perhaps simply the edge 
of clearing on an otherwise cluttered surface of his desk), set 
it down on the left and continue to move it to the right. What 
is needed is a way to keep the motion indicating signals from 
undergoing spurious behavior during this maneuver, so that 
the pointer on the screen behaves in an expected and 
non-obnoxious manner. The function of the "hold" button 
may be performed automatically by a proximity sensor on 
the underside of the mouse that determines that the mouse is 
not in contact with the work surface, or by noticing that all 
or a majority of the pixels in the image have "gone dark** 
(it's actually somewhat more complicated than that — we 
shall say more about this idea in the next paragraph). 
Without a hold feature, there may be some slight skewing of 
the image during the removal and replacement of the mouse, 
owing either: (a) to a tilting of the field of view as the mouse 
is lifted; or (b) to some perverse mistake where frames for 
two disparate and widely separated spatial features imaged 
at very different times during the removal and replacement 
are nevertheless taken as representing a small distance 
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between two frames for the same feature. Aconvenient place 
for an actual hold button is along the sides of the mouse near 
the bottom, where the thumb and the opposing ring finger 
would grip the mouse to lift it up. A natural increase in the 
gripping force used to lift the mouse would also engage the 
hold function. A hold feature may incorporate an optional 
brief delay upon either the release of the hold button, 
detection of proper proximity or the return of reasonable 
digitized values. During that delay any illumination control 
servo loops or internal automatic gain controls would have 
time to stabilize and a new reference frame would be taken 
prior to the resumption of motion detection. 

And now for this business of the pixels in the image 
"going dark". What happens, of course, is that the IR light 
from the illuminating LED no longer reaches the photo 
detectors in the same quantity that it did, if at all; the 
reflecting surface is too far away or is simply not in view. 
However, if the seeing eye mouse were turned over, or its 
underside exposed to an intensely lit environment as a result 
of its being lifted, then the outputs of the photo detectors 
might be at any level. The key is that they will be uniform, 
or nearly so. The main reason that they become uniform is 
that there is no longer a focused image; all the image 
features are indistinct and they are each spread out over the 
entire collection of photo detectors. So the photo detectors 
uniformly come to some average level. This is in distinct 
contrast with the case when there is a focused image. In the 
focused case the correlations between frames (recall the one 
over, one over and one down, etc.) exhibit a distinct phe- 
nomenon. 

Assume that the spatial features being tracked mapped 
exactly onto the photo detectors, through the lens system, 
and that mouse movement were jerky by exactly the amount 
and in the directions needed for a feature to go from detector 
to detector. Now for simplicity assume also that there is only 
one feature, and that its image is the size of a photo detector. 
So, all the photo detectors but one are all at pretty much the 
same level, and the one detector that is not at that level is at 
a substantially different level, owing to the feature. Under 
these highly idealized conditions it is clear that the corre- 
lations will be very well behaved; eight "large*' differences 
and one small difference (a sink hole in an otherwise fairly 
flat surface) in a system using nine trials for a nearest 
neighbor algorithm (and remembering that there may have 
been no motion). [Note: The astute reader will notice that the 
"large" difference in this rather contrived example actually 
corresponds to, or originates with, only one pixel, and 
probably does not deserve to be called "large" — recall the 
earlier shifted film analogy. The only light passed by the 
films for this example would be for the one pixel of the 
feature. A more normal image having a considerably more 
diverse collection of pixels increases the difference to where 
it truly is a "large" difference.] 

Now, such highly idealized conditions are not the usual 
case. It is more normal for the image of the tracked spatial 
features to be both larger and smaller than the size of the 
photo detectors, and for the mouse motion to be continuous, 
following a path that allows those images to fall onto more 
than one detector at once. Some of the detectors will receive 
only a partial image, which is to say, some detectors will 
perform an analog addition of both light and dark. The result 
is at least a "broadening" of the sink hole (in terms of the 
number of photo detectors associated with it) and very 
possibly a corresponding decrease in the depth of the sink 
hole. Hie situation may be suggested by imagining a heavy 
ball rolling along a taut but very stretchable membrane. The 
membrane has a discrete integer Cartesian coordinate sys- 
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tem associated with it. How much does the membrane and navigation arrangement 1 that is generally of the type 

distend at any integer coordinate location as the ball rolls? described by the incorporated Patents. An LED 2, which 

First imagine that the ball is of a very small diameter but may be an IR LED, emits light which is projected by lens 3 

very heavy, and then imagine that the ball is of a large (which instead of being separate may be an integral part of 

diameter, but still weighs the same. The analogy may not be 5 me LED's package), through orifice 13 in bottom surface 6 

exact, but it serves to illustrate the idea of the "sink hole" and onto a re 8 ioQ 4 that * P art of a work surface 5. The 

mentioned above. The general case is that the generally flat average angle of incidence is preferably within the range of 

surface with sharply defined sink hole becomes a broad fi ^. to twenty degrees. Although it has been omitted for 

concavity, or bowl. clarity, the orifice 13 might include a window that is 

w u n * *u -e a jx j •« j i ,i transparent for the light from LED 2, and which would serve 

We shall term the surface produced or described by the 30 r 6 . ' . 

, , £ , c „ , .„ to keep dust, dirt or other contamination out of the innards 

various correlation values the correlation surface and will, c r . .... - 

. t . t j • ■ ■ of the seeing eye mouse. Work surface 5 might belong to a 

at various times, be most interested in the shape of that • i u- ♦ u a n u ■« 

surface special object, such as a mouse pad, or more generally, it will 

S ,,! Ce * „ * . . , . ^. not > and might be the surface of nearly anything except 

We say all of this to make two points. First, the shifting smooth glasSi Examples of materials include, but 

shape of the concavity in the correlation surface as the 55 are not UndM to> paper> dothf laminated plastic topSf 

seeing eye mouse moves allows interpolation to a granular- painted sur f aces , frosted glass (smooth side down, thank 

ity finer than the simple size/spacing of the photo detectors. you)> desk padSf real wood> fake woodj etc GeneraU v any 

We point this out, with the remark that our seeing eye mouse micro textured surface having features whose size falls 

can do that, and leave it at that. The full details of interpo- within tne range of 5 t0 500 m i cr0 ns will do. 

lation are described in the incorporated Patents. No further ^ illumination of micr0 textured surfaces is most effec . 

dKcussion of interpolation is believed necessary. Second, ^ when done from me sid ^ ^ accentuates tDe ttera 

and this u our real reason for the discussion of the preceding of afld shadows ^ b surface * 

paragraphs, is the observation that what happens when the Suitable M ^ cs of incidence for iUuminat f on 

seeing eye mouse is picked up is that the concavity in the coyer lhe of abom fiye lQ d A 

correlation surface goes away, to be replaced by generally » ^ ^ ^ ^ fa J ^ J 

equal values for the correlations (i.e., a fiat correlation lished) havin sim k variatioQS in reflectivit 7 owing t0 

surface). It is when this happens that we may say with 5 . ' , x „7L i • „• 1 . » °, 

. ' . . . f . (micro scale) compositional vanation works, too. In such a 

considerable assurance that the seeing eye mouse is air /a * . ■* l . JX lL t r 

. iL A „ • , l u i j r case (and assuming that it can be guaranteed) the angle of 

borne, and can then automatically invoke the hold feature, • • ^ <- .„ _ - , , 7 u - * 

' , t . , ./. . . ' in incidence for the illumination could approach ninety 

until after such time that a suitable concavity ( bowl ) JU , - M , , . u j 

7 v 7 degrees, since the urge to create shadows goes away. 

reappears. However, such a smooth yet micro detailed surface is not 

Another method for invoking or initiating a hold feature what we would ordinarily think of when we say "arbitrary 

is to simply notice that the seeing eye mouse is moving surface" , and a seeing eye mouse intended for use on an 

faster than a certain threshold velocity (and is thus presum- ^ "arbitrary surface" that is more likely micro textured would 

ably experiencing an abrupt retrace motion in a maneuver work best if equipped to provide a grazing angle of incident 

intended to translate the screen pointer further than the illumination 

available physical space within which the mouse is An image of me mummated region 4 is projected through 

operating) Once the velocity threshold is exceeded the an tical wMow 9 m acka rtion Sa of an mlegrated 

motion indicating signals mat would otherwise be associated circuit and onto m 1Q of ^ detectors . ^ ^ done 

with that movement are suppressed until such time as the ^ tne aid of lens 7 The pack tion 8fl mi ht alsQ 

velocity drops below a suitable level. dispense ^ sepafate window 9 ^a lens 7 by combining 

BRIEF DESCRIPTION OF THE DRAWINGS tnei n into one and the same element. The photo detectors 

™„ . * j • c may comprise a square array of, say, 12 to 24 detectors on 

FIG. 1 is a simplified pictographic cut-away side view of a sidCj eacfl de(ector ^ a hoto whose hotQ 

a pnor art imaging and navigation arrangement; regi(m fa 45 by 45 microns ^ of 6Q 

FIG. 2 is a bottom view of a mouse constructed in center t0 center spacing. The photo transistors charge 

accordance with the invention; capacitors whose voltages are subsequently digitized and 

FIG. 3 is a side perspective view of a mouse constructed stored in a memory. The array 10 is fabricated onto a portion 

in accordance with one aspect of the invention; and 50 of an integrated circuit die 12 affixed by an adhesive 11 onto 

FIG. 4 is a simplified side cut-away view of a proximity package portion Sb. What is not shown are any of the details 

sensor in the base of the mouse of FIGS. 2 and 3 and used of how the integrated circuit is held in place (probably by a 

to automatically activate a hold feature; printed circuit board), the shape or composition of the 

FIG. 5 is a simplified flow chart describing an aspect of lenses, or of how the lenses are mounted; it is clear that those 

internal seeing eye mouse operation related to the operation 55 things are doable in a conventional manner. It is also clear 

of the hold feature when used in conjunction with a feature mat tDe general level of illumination of region 4 may be 

called prediction; controlled by noticing the output levels of the photo detec- 

FIG. 6 is a simplified portion of a modification of the flow tors ™ d adjusting the intensity of light issuing from the LED 

chart of FIG. 5 and illustrates the velocity detection method 2 * be eUher continuous control or pulse width 

of invoking the hold feature; and 60 modulation, or some combination of both. 

FIG. 7 is a perspective view of a plotted correlation 0nce a 8 ain » the reader fa reminded that the details of the 

surface that has good concavity. motion sensing operation are thoroughly described in the 

incorporated patents (and briefly described in the Summary); 

DESCRIPTION OF A PREFERRED accordingly, they need not be repeated here. 

EMBODIMENT 65 Refcr now tQ F , G 2 which fa a boUom of a mouse 

Refer now to FIG. 1, wherein is shown a simplified 14 constructed in accordance with the invention. In short, 

representation of a cut-away side view of a prior art imaging this bottom view of this particular seeing eye mouse 14 
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looks very similar to the bottom view of a particular con- is flat, or nearly so. Rather than separately detecting uniform 

ventional mouse from Hewlett-Packard Co., to wit: the levels (which would use hardware not otherwise present), 

C1413A. The major difference is that where there ought to we prefer instead to examine the shape of the correlation 

be a ball there is a protective lens or window 16 that is surface, (which surface we need for other reasons, anyway), 

transparent to IR light. This is the omitted transparent 5 T° e most probable cause of a flat correlation surface is that 

window in orifice 13 that was mentioned in the description me m °use has been picked up. This mode of operation may 

of FIG. 1. Also missing is the usual rotatable annulus that require that there be a fairly narrow depth of field, lest there 

serves as a removable retainer to allow access to the ball for occur dela y m activating the hold. Such delay could 

cleaning or replacement. What is shown in the figure is the produce minor artifacts in screen pointer movement. These 

underside 15 of the mouse 14 (corresponds to 6 in FIG. 1), 10 mi S ht include slight unintended screen pointer movements 

low friction glides 19 and connecting cable 17 with its strain owin g t0 of the mouse as it is either picked up or 

relief 18. Of course, our seeing eye mouse 14 could be a replaced. As long as activating the hold feature (however 

cordless mouse, as well, with an optical or radio communi- done * whether manually or automatically) forces acquisition 

cation link to the computer. °f a new reference frame before resuming the production of 

Refer now to FIG. 3, wherein is shown a side perspective is mo,ion si 8 n j ds ' . there sh , ould 00 dan S er * P"*"** ■ 

view of a mouse 14 constructed in accordance with one ^punous indicahon resulting from the combination of old 

aspect of the invention. That aspect of the invention is the data ^ new data ^ ^ accidentally looks like a 

hold feature. The hold feature is an aspect of seeing eye P roper sma " motlon ln f ^appropriate direction, 

mouse operation that suspends the production of movement ""TV' ™* mere ™f™ , level detection (of, say, a 

information or signals to the computer when it is determined 20 *™Pk &ame) it may be difficult to guarantee that while m 

that the mouse is not suitably proximate to the work surface m f oh ° n . *" me Mr h , e / e , occu r i no °f ^ °*f* 

whose spatial features are being tracked. This allows the ° f a br, 6 h ! «"«*> w ° m the algorithm. It wdl 

seeing eye mouse to be picked up, translated and set back be W««ated that the shape of the correlation surface is a 

down! or, as we shall term such an operation, "swiped" much ™re "liable mdicator. All of that said, it must still be 

across the work surface. 25 remembered that the steering, as it were, of the screen 

pointer is an incrementally driven servo-like operation per- 

In particular, the seeing eye mouse 14 in FIG. 3 includes formed by a humaa ^ tf me intef fen , t there 

at least one hold button 24 located in side skirt 20 near the ^ just keep moving ^ mouse ^ needed , SmaU f _ 

bottom surface 15 so as to be beneath the right thumb or the bations during swiping are not fatalj and ma QOt even be 

left ring finger, depending upon which hand the operator is particularly noticeable, depending upon the specific appli- 

using. There may be another symmetrically located button cation being per formed 

on the other side (not shown) that would contact either the Refer now to FIG. 5, wherein is shown a flow chart 29 that 

e um or e rig rmg nger. describes an aspect of seeing eye mouse operation involving 

The mouse 14 conventionally includes a surface 21 which me hold and prediction properties. We may assume that there 

nestles in the palm of the hand, and first and second 35 is some start con^ion or location 30, from which is reached 

"regular" mouse buttons 22 and 23 that are actuated by the step 31: ACQUIRE A REFERENCE FRAME. This refers to 

index and middle fingers. These operate in their normal illuminating the LED 2 and storing a collection of digitized 

fashion. photo detector values into an array of memory (not shown). 

Button or buttons 24 are activated by a natural increase in The next step 32 is ACQUIRE A SAMPLE FRAME. This 

the gripping force needed to pick the mouse 14 up during a ^ refers to the same actions, except that the data is stored in a 

swipe. When one or both of these button are pressed the hold different array of memory, and may reflect mouse motion 

feature is activated. For the duration of the hold the sending relative to where it was when step 31 was performed. At step 

of motion signals to the computer is suspended. When the 33, COMPUTE CORRELATION VALUES, the nine (or 

hold is over (the buttons are released) a new reference frame perhaps twenty-five) correlation values are quickly com- 

is taken before any new motion signals are sent to the 45 puted by some heavy duty dedicated arithmetic hardware 

computer. This allows swiping, and has the advantage that assisted by automatic address translation and a very wide 

the user has the ability to expressly force the onset of the path out of the memory arrays. At step 34, IS THE COR- 

hold feature. RELATION SURFACE SUITABLY CONCAVE?, the 

The hold feature could also be automatically activated by nature of the correlation surface described by the collection 

the action of a separate proximity sensor on the bottom of 50 of correlation values computed in step 33 is examined. We 

the mouse. This is what is shown in FIG. 4, where a want to know if it is shaped like a bowl, and if so, "how 

shouldered aperture 26 in the base 6 receives a shouldered much water will it hold," so to speak, 

plunger 25 made captive by the lever arm of a switch 28 If the shape of the correlation surface is a good bowl, then 

above. The switch 28 is activated by movement of the path 36 takes us to the optional step 37: IS THE HOLD 

plunger 25, such that when the plunger moves significantly 55 BUTTON PRESSED?; more about that in the next para- 

in the direction of arrow 27 the hold feature is activated. The graph. Otherwise, we have a flat correlation surface, or a 

exact nature of the separate proximity sensor is a matter of "bad bowl," and proceed along path 35 to optional step 42, 

choice, and while it could be a simple as the micro switch DELAY There are several possible causes for this exit from 

28 operated by the weight of the mouse through the plunger qualifier 34: e.g., extreme velocity, a suddenly featureless 

25, other, non-mechanical, methods are also possible. 60 work surface, and, an airborne mouse. In the absence of an 

Yet another way to automatically activate and deactivate explicit HOLD button, we will rely upon exit path 35 to 

the hold feature is to examine the nature of the digitized data provide proper seeing eye mouse behavior by suppressing 

of the array 10 of photo detectors. When the outputs of the motion signals to the computer during the airborne portion 

photo detectors become sufficiently uniform it may be of a swiping operation. 

surmised that there is no longer an image with variations 65 If the seeing eye mouse does have a HOLD button, then 

projected onto the array 10 of photo detectors. This unifor- optional qualifier 37 is present, and it is there that the status 

mity will reveal itself by producing a correlation surface that (pressed or not) of the HOLD 24 button is determined. The 
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case where it is pressed is treated the same as that for a bad 
bowl at qualifier 34. That is, path 38 is taken, which also 
leads to optional step 42. 

Optional step 42 provides a delay which may be useful in 
several ways. First, if there is a swipe in progress, then it 5 
takes some time, and by not imagining during that time some 
battery power can be saved. Also, suppose that the nature of 
the delay is slightly more complex than a pause in the 
motion of a moving finger on the flow chart. Suppose that 
the ACQUIRE REFERENCE FRAME step 31 were influ- 10 
enced by there having been a delay at step 42, in that part 
way through the delay an illumination level control opera- 
tion is initiated. This could allow time for re-adjustment of 
illumination levels, and so forth. Whether or not there is a 
DELAY at optional step 42, path 43 leads back to step 31, is 
where another motion detection cycle begins. 

To resume, path 39 leads to step 40: PREDICT SHIFT IN 
REFERENCE FRAME. As mentioned above, it is generally 
not necessary to obtain and maintain actual velocities in X 
and Y, and time interval information, to find the displace- 20 
ment needed for prediction. One can imagine measurement 
environments where that might be needed, but the one 
shown here is not one of them. Instead, the predicted shift 
can be taken as the amount of movement corresponding to 
the correlation at the preceding step 34. 25 

The next step 44 is OUTPUT AX & AY. It is here that we 
take note of how much mouse motion there has been since 
the last measurement cycle. The amount of shift needed to 
attain correlation is the desired amount. These values may be 
found by noticing which comparison frame actually corre- 
lated (assuming no interpolation). These "raw" AX and AY 
motion values may be accumulated into running values that 
are sent to the computer at a lower rate than that at which the 
raw values of step 44 are produced. 

At qualifier 45 we ask if we NEED A NEW REFERENCE 3 
FRAME?. If the answer is YES, then path 46 leads to step 
48: STORE PRESENT SAMPLE FRAME IN REFER- 
ENCE FRAME. (A little thought will confirm that this 
re-use of the sample frame cooperates with not having to 4Q 
maintain actual velocities and time intervals for the predic- 
tion process. If we took a separate new reference frame it 
would complicate a lot of things, and would probably force 
the use of D=RT — i.e., the distance formula — for 
prediction.) 45 

We need a new reference frame when there has been 
enough shifting of it, owing to predictions, that not enough 
of it overlaps the comparison frames for reliable correla- 
tions. Somewhere in the range of three to five shifts (that do 
not retrace themselves) is about the limit for a 16x16 50 
reference frame. 

If the answer to qualifier 45 is NO, and we do not need to 
replace the reference frame, then path 47 takes us to the 
same step 49 as does the path leading from step 48. Step 49, 
SHIFT REFERENCE FRAME, performs the actual perma- 55 
nent shift of the values in the memory array representing the 
reference frame. The shift is by the prediction amount, and 
data shifted away is lost. Following the shifting of the 
reference frame path 50 returns to step 32, ACQUIRE A 
SAMPLE FRAME, where the next measurement cycle eo 
begins. 

Refer now to FIG. 6, wherein is shown a simplified flow 
chart segment 50 that shows how to replace step 44 of the 
flow chart 29 in FIG. 5 with steps 51-55. The purpose for 
doing this is similar to the various manners of hold operation 65 
already described, and may be used in conjunction 
therewith, or instead thereof. The general idea of the modi- 
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fication represented by FIG. 6 is to outfox the computer by 
either not sending any updated information by skipping step 
55Aor (optionally, with step 55B) sending zeros for AX and 
AY, even when that is not true. This is done whenever step 

52 ascertains that the rate of mouse motion exceeds, say, 
three to six inches per second. For a given seeing eye mouse 
such a limit is easily expressed as a displacement by a 
certain number of pixels within some number of measure- 
ment cycles, assuming that the measurement cycle rate is 
fast compared to normal mouse motion. The idea is that 
normal casual mouse motion probably will not require either 
a new nearest neighbor reference frame (let alone a maxi- 
mally shifted one for 5x5 near neighbor operation) every 
measurement cycle for some large (say, ten to twenty-five) 
number of consecutive measurement cycles. For if that were 
the case, the seeing eye mouse would be operating on the 
hairy edge of the hold mode via a NO answer to qualifier 34 
and path 35. (According to the assumption, any higher 
velocity will result in loss of correlation!) That is, the 
expectation is that taking a new reference frame is normally 
much less frequent. Of course, it could happen that the 
velocity of the mouse is really high, and path 35 gets used, 
anyway. That is as it should be. But if the measurement cycle 
rate is not sufficiently high with respect to normal expected 
mouse motion, then it might not be appropriate to use the 
technique of FIG. 6. 

Step 51 represents anything in the old step 44 over and 
above the actual communication to the computer of the 
values AX and AY. A tricky example of this difference might 
be an internal accumulation of motion that has not yet be 
dispatched to the computer, owing to a higher internal 
motion measurement cycle rate for the seeing eye mouse 
than the information exchange rate with the computer. Now, 
it may well be the case that in some systems this accumu- 
lated information is used for internal mouse purposes other 
than strictly for keeping the computer informed. If so, then 
it would need to be preserved, for all that qualifier 52, path 

53 (and bypassed step 55A) need to accomplish is NOT tell 
the computer there has been motion; we want to fool the 
computer but without making the mouse lose its mind. 

It will be noticed that if such an accumulation were 
allowed to continued during a rapid retrace, intended to 
mimic picking the mouse up, the computer might still win in 
the end when the velocity drops to normal amounts and the 
accumulation is finally sent; the screen cursor could snap to 
the correct location, anyway, depending upon how the 
overall system works. In such a case a separate set of 
accumulations should be maintained, with those for the 
computer remaining in bypassed step 55A. 

Of course, it may be the case that there is no internal use 
by the mouse of accumulated AX and AY, other than to send 
it to the computer. In that case nothing needs to be done, 
other than to leave that accumulation in bypassed step 55A. 
It is also possible that in the mouse there simply are no 
accumulations to cause such concerns; say, any such accu- 
mulations were done by software in the computer. 

Finally, refer now to FIG. 7. It is a plot 56 of a near 
neighbor (5x5) correlation surface 57 having a suitable 
concavity. The two horizontal axes 58 and 59 represent the 
X and Y axes of mouse motion; the units indicated along the 
axes are pixels. Drawn onto the plane of the axes 58 and 59 
are smoothed and interpolated contour lines 60 intended to 
further indicate the shape of the correlation surface directly 
above. The vertical axis 61 a measure of correlation 
expressed in essentially arbitrary units. 

We claim: 

1. A hand held pointing device for a computer system, the 
pointing device comprising: 
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a housing having a bottom surface that moves against a 

work surface having imageable features; 
the housing also having a top surface shaped to receive the 

human hand; 

the housing also having a skirt connecting a perimeter of 5 
the bottom surface with tie top surface; 

the housing also having a first axis extending generally in 
the direction from where the heel of the hand rests on 
the top surface to where the middle finger rests on the 
top surface, and a second axis perpendicular to the first, 
both axes being parallel to the bottom surface; 

an aperture in the bottom surface; 

a source of illumination mounted within the interior of the 
housing, proximate the aperture and that produces is 
highlights and shadows by illuminating micro texture 
on the work surface; 

a lens mounted within the interior of the housing, proxi- 
mate the aperture and receiving light from the illumi- 
nated micro texture; 20 

an optical motion detection circuit mounted within the 
interior of the housing and optically coupled by the lens 
to the highlights and shadows of the illuminated micro 
texture, the optical motion detection circuit producing 
motion signals indicative of motion in the directions 
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along the first and second axes and relative to the 
highlights and shadows of the illuminated micro tex- 
ture; and 

wherein the optical motion detection circuit comprises a 
plurality of photo detectors each having an output, a 
memory containing reference and sample frames of 
digitized photo detector output values representing the 
highlights and shadows of the illuminated micro tex- 
ture on the work surface; 

a proximity detector that detects when the bottom surface 
is away from the work surface by more than a selected 
distance, that is coupled to the optical motion detection 
circuit, and that inhibits production of the motion 
signals when the bottom surface is away from the work 
surface by more than the selected distance, the prox- 
imity detector including arithmetic comparison circuits 
coupled to the reference and sample frames in the 
memory of the optical motion detection circuit, the 
proximity detector detecting an amount of correlation 
between the reference frame and sample frames that is 
less than an amount of correlation corresponding to the 
selected distance. 

***** 
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